Analysis and Determination of Inner Lip texture Descriptors for Visual Speech Representation

The problem of visual speech representation for bimodal based speech recognition includes particular challenges in the modeling of the inner lip texture reflecting different pronunciations, such as the appearance of teeth and tongue. This paper proposes and analyzes several possible statistical inner lip texture descriptors to determine an effective and discriminant feature. Simply using grayscale without full specification of the underlying colour model tends to loss some significant discriminative information. Therefore thorough exploration on the color space components selection in computing the local inner lip texture is thus a primary goal of the present research. The L channel of Lab color space is finally determined as the basis for the development of the inner lip texture model. Through feature level fusion, the final classification of visual speech is performed based on the proposed inner lip texture descriptor and standard geometric features. Together with audio speech, this paper furthers the development of the CHMM based bimodal Chinese character pronunciation recognition system. The experimental results show that the local inner texture descriptors, such as the color moment with geometric feature, outperform the holistic inner texture descriptors, such as the statistical histogram, in representing visual speech with the close discriminability but low dimensionality.